Implement a basic generic API client #313

burnash · 2023-12-29T15:06:47Z

Tell us what you do here

This PR implements a basic generic API client. Mostly by trying to generilize pagination and auth logic which many of the sources in this repo reimplement. The goal is to speed up sources development by unifying some common functionality. At the same time I'm trying to design the client's API to be as close to Requests as possible.

Prior relevant work: dlt-hub/dlt-init-openapi-demo#2

Example: Get issues GitHub resource from dlt docs.

Original

import dlt
from dlt.sources.helpers import requests

@dlt.resource(
    table_name="issues",
    write_disposition="merge",
    primary_key="id",
)
def get_issues(
    updated_at=dlt.sources.incremental("updated_at", initial_value="1970-01-01T00:00:00Z")
):
    # NOTE: we read only open issues to minimize number of calls to the API. There's a limit of ~50 calls for not authenticated Github users
    url = f"https://api.github.com/repos/dlt-hub/dlt/issues?since={updated_at.last_value}&per_page=100&sort=updated&directions=desc&state=open"

    while True:
        response = requests.get(url)
        response.raise_for_status()
        yield response.json()

        # get next page
        if "next" not in response.links:
            break
        url = response.links["next"]["url"]

pipeline = dlt.pipeline(
    pipeline_name="github_issues_merge",
    destination="duckdb",
    dataset_name="github_data_merge",
)
load_info = pipeline.run(get_issues)
print(load_info)

Using RESTClient

import dlt
from rest_api import RESTClient

github_client = RESTClient(base_url="https://api.github.com")


@dlt.resource(
    table_name="issues",
    write_disposition="merge",
    primary_key="id",
)
def get_issues(
    updated_at=dlt.sources.incremental(
        "updated_at", initial_value="1970-01-01T00:00:00Z"
    )
):
    for page in github_client.paginate(
        "/repos/dlt-hub/dlt/issues",
        params={
            "since": updated_at.last_value,
            "per_page": 100,
            "sort": "updated",
            "direction": "desc",
            "state": "open",
        },
    ):
        yield page


# The rest of the pipeline remains the same
pipeline = dlt.pipeline(
    pipeline_name="github_issues_merge",
    destination="duckdb",
    dataset_name="github_data_merge",
)
load_info = pipeline.run(get_issues)
print(load_info)

rudolfix · 2023-12-30T11:43:58Z

@burnash

This is very helpful. At least from my side it makes way clearer how to design code here. The idea of request-like helper that has pagination, auth and other things built-in is good.

api
basically I see two components

paginated requests which has similar interface to requests but instead of returning Response it retruns a generator, yielding pages of data

on top of usual requests arguments it also accepts

authenticator instance (we already have auth argument)
paginator instance
response unwrapper (extracts page of data from response)
incremental instance if endpoint is incremental
incremental param to connect incremental instance to

the returned generator may be passed directly to dlt.resource (I think it will be generator function, not generator so you can bind incremental to it)

ie.

paginated = paginated_request(url, pagination=json_paginator(), auth=bearer(api_secret), incremental=dlt.source.incremental("updated_at"), incremental_param="sice")

resource = dlt.resource(paginated.data, name="...", primary_key="...")

where paginated.data is a generator function ie.

def data(self, last_value: dlt.source.incremental[Any] = None):
	... # main loop happens here

I'd strive move as much logic into a common code and make paginators and authenticators as simple as possible. For example paginator could get a last response and a prepared request and modify it to get next page. we could even make it more abstract and pass header, paramas etc.

declarative part. it is similar to Alex to EndpointFactory but also adds definitions of the endpoints. why we need it:

to declare most common behavior. we hope 90% of apis can be just declared using starndard auth and paginators
to easily add new endpoints
to further ease using of incremental

To declare an endpoint we basically provide kwargs to the paginated_request. the declaration can also take default authenticator and paginator

then in the dlt.source user can enumerate all paginated_requests and create dlt.resources out of them. there should be also a method to get a dynamic endpoint out of it (then it works like Alex's factory)

Approach
Currently you try to fit a tool into existing code. So you get those complicated conditional paginators serving many different endpoint types. All try something different: let's build a decent API and then fit our verified sources. starting with simple stuff like pokemon and then going for zendesk..

sources/api_client.py

sultaniman

Looks cool, I also wonder if pagination strategy should also take params from .dlt/config.yml?

sources/api_client.py

sultaniman · 2024-01-24T13:17:08Z

sources/api_client.py

+    def get(self, path="", params=None):
+        return self.make_request(path, method="get", params=params)
+
+    def post(self, path="", json=None):


maybe we can also use typing override to define signature variances?

@sultaniman could you elaborate on this one?

sources/api_client.py

sources/zendesk/helpers/paginators.py

sources/notion/__init__.py

sources/rest_api/client.py

sources/rest_api/detector.py

sources/rest_api/client.py

sources/rest_api/detector.py

sources/rest_api/client.py

…s client

burnash · 2023-12-29T15:44:07Z

tests/notion/test_notion_database.py

@@ -3,7 +3,7 @@
 from sources.notion.helpers.database import NotionDatabase
 from sources.notion.helpers.client import NotionClient

-
+@pytest.mark.skip


Skipping this since it's testing the former client

burnash · 2023-12-29T15:45:20Z

tests/personio/test_personio_source.py

@@ -21,7 +21,7 @@ def test_all_resources(destination_name: str) -> None:
    table_counts = load_table_counts(pipeline, *table_names)

    assert table_counts["employees"] >= 31
-    assert table_counts["absence_types"] >= 6
+    assert table_counts["absence_types"] >= 5


Adjusting for a fresh Personio account.

burnash · 2023-12-29T15:52:14Z

sources/api_client.py

+        self.base_url = base_url
+        self.headers = headers
+        self.auth = auth
+        self.paginator = paginator if paginator else HeaderLinkPaginator()


Not sure what should be a default here, so I went with a personal preference.

sources/api_client.py

sources/rest_api/client.py

sources/rest_api/__init__.py

burnash · 2024-03-12T20:49:14Z

sources/rest_api/typing.py

+)
+
+PaginatorConfigDict = Dict[str, Any]
+PaginatorType = Union[Any, BasePaginator, str, PaginatorConfigDict]


Fixed in ba46fda by dlt-hub/dlt#1082
Thanks again @sultaniman for the review!

sources/rest_api/client.py

rudolfix

TODO (for later)

"Endpoint" does not make it clear if single entity is returned or a list of entities. should we have "none" paginator to indicate that? or some boolean flag
What is the use of SinglePage paginator now? we have this add_limit which makes any paginator single page
I should be able to bind same incremental to start/since param (start_value) and to end/before param (end_value).
I should be able to bind same transformer to many params when id is compound (this is less important)
paginators should be configspecs (dataclasses) so they can be configured and initialized like authentiators. we could use the same factory approach for both - I can do this one
RESTAPIConfig should accept an instance of DltResource here to allow completely code to be mixed with declared endpoint resources

resources: List[Union[str, EndpointResource]]

(this is less important)
7. several types in typing.py must go to dlt core ASAP. I can also handle that. we could also generate typed dicts automatically for all our configspecs.

sources/rest_api/__init__.py

rudolfix · 2024-03-14T12:54:37Z

sources/rest_api/__init__.py

+                param_name=param_name,
+                field_path=resolved_param.resolve_config.field_path,
+            ):
+                items = items or []


we should allow for items that are not lists and encapsulate them. also "items" that evaluate to False - that rather should not happen

sources/rest_api/client.py

sources/rest_api/__init__.py

sources/rest_api/config_setup.py

sources/rest_api/detector.py

sources/rest_api/paginators.py

sources/rest_api/config_setup.py

rudolfix

LGTM!

sources/rest_api/config_setup.py

burnash marked this pull request as draft December 29, 2023 15:07

burnash requested a review from rudolfix December 29, 2023 15:08

sultaniman reviewed Jan 24, 2024

View reviewed changes

sources/api_client.py Outdated Show resolved Hide resolved

sultaniman reviewed Jan 24, 2024

View reviewed changes

francescomucio reviewed Jan 24, 2024

View reviewed changes

sources/notion/__init__.py Outdated Show resolved Hide resolved

sultaniman reviewed Feb 5, 2024

View reviewed changes

sources/rest_api/client.py Outdated Show resolved Hide resolved

sources/rest_api/detector.py Outdated Show resolved Hide resolved

sources/rest_api/detector.py Outdated Show resolved Hide resolved

sultaniman reviewed Feb 12, 2024

View reviewed changes

sources/rest_api/client.py Show resolved Hide resolved

sultaniman reviewed Feb 12, 2024

View reviewed changes

sources/rest_api/detector.py Show resolved Hide resolved

sultaniman reviewed Feb 12, 2024

View reviewed changes

sources/rest_api/client.py Show resolved Hide resolved

burnash added 15 commits February 19, 2024 18:21

Implement a basic generic API client; convert some sources to use thi…

7d65701

…s client

Move the paginaton loop into APIClient

f7af81b

Factor out common code

4bfdc13

Refactor common code

3ce56c1

add generic rest source and an example pipeline

ce784d4

Restructure the REST client add pagination detector

8d7d0bd

fix the paginator detector

8bde92a

Accept paginator instance

4b686a0

Add Offset paginator

2e297e2

Add comments

f5db3aa

Factor out resources

650c3ca

Add Literal

e61e304

Remove the example

5e63d33

Add logging

d55ae29

Handle depended resources

01fe721

burnash commented Mar 13, 2024

View reviewed changes

burnash added 4 commits March 13, 2024 18:15

Refactor configuration setup into a dedicated module

94352ed

Move response hooks setup and handling out of RESTClient

70496ec

Remove unused imports

2e3d50a

Fix hooks typing

b6d3794

burnash requested a review from rudolfix March 14, 2024 10:39

rudolfix requested changes Mar 14, 2024

View reviewed changes

burnash and others added 14 commits March 20, 2024 13:26

Rename args of the OffsetPaginator

3b7a0b6

Create a RESTClient per resource

b971f9b

Handle both error statuses and response actions

ebff65c

Initial version of the README.md (#389)

4fb4ce1

Remove commented code

59bba2e

Clean up docstrings

390c233

Remove the useless conditional init for items

5ecc426

Fix grammar in the README

6924233

Remove UnspecifiedPaginator

ff5ddd8

Format README

c24a5aa

Add handling end_value and end_param

d63e30f

Use NamedTuple for incremental params

5cbf863

Move check_connection to utils

6b7a891

Instantiate auth based on type

6dab451

burnash commented Mar 22, 2024

View reviewed changes

sources/rest_api/config_setup.py Show resolved Hide resolved

burnash and others added 5 commits March 22, 2024 13:30

Merge branch 'master' into enh/api_helper

32e8327

Update lock file

559f3a2

Sthor/api helper updates (#400)

cf8e266

Remove unused imports

b8e6b5d

Use jsonpath for next_path; remove create_nested_accessor

e6d927c

rudolfix approved these changes Mar 22, 2024

View reviewed changes

sources/rest_api/config_setup.py Show resolved Hide resolved

burnash merged commit 079a59f into master Mar 22, 2024
14 checks passed

burnash mentioned this pull request Mar 24, 2024

Add RESTClient dlt-hub/dlt#1141

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a basic generic API client #313

Implement a basic generic API client #313

burnash commented Dec 29, 2023 •

edited

Loading

rudolfix commented Dec 30, 2023

sultaniman left a comment

sultaniman Jan 24, 2024

burnash Feb 5, 2024

burnash Dec 29, 2023

burnash Dec 29, 2023

burnash Dec 29, 2023

burnash Mar 12, 2024

rudolfix left a comment

rudolfix Mar 14, 2024

rudolfix left a comment

Implement a basic generic API client #313

Implement a basic generic API client #313

Conversation

burnash commented Dec 29, 2023 • edited Loading

Tell us what you do here

Example: Get issues GitHub resource from dlt docs.

Original

Using RESTClient

rudolfix commented Dec 30, 2023

sultaniman left a comment

Choose a reason for hiding this comment

sultaniman Jan 24, 2024

Choose a reason for hiding this comment

burnash Feb 5, 2024

Choose a reason for hiding this comment

burnash Dec 29, 2023

Choose a reason for hiding this comment

burnash Dec 29, 2023

Choose a reason for hiding this comment

burnash Dec 29, 2023

Choose a reason for hiding this comment

burnash Mar 12, 2024

Choose a reason for hiding this comment

rudolfix left a comment

Choose a reason for hiding this comment

rudolfix Mar 14, 2024

Choose a reason for hiding this comment

rudolfix left a comment

Choose a reason for hiding this comment

burnash commented Dec 29, 2023 •

edited

Loading